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-  Synchronic 
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Justification 





An Old Mantra 


Data Collection: 
- One data point is good, but more data points 
are better! 
Bootstrapping: 
- One sample is good, but more samples are 
better! 
Modeling: 
- One model is good, but more models are 
better! 


Model Composed of Other Models 


Justification 


- Building models beyond the first is good for 
comparison's sake. 

- But we can also combine models together to 
form new models. 





Stacking 





Different Models, Same Data 


e Similar to a bagging approach, 
stacking is a form of averaging 

e Unlike a bagging approach, stacking 
typically uses the same training 
data for every model 

e The innovation of stacking is the 
use of different kinds of models 


Stacking 


Meta-Classifier/Meta-Regressor 


e First, we ask several different 
models to make predictions about 
the target 

e Rather than taking a simple average 
or vote to determine the outcome, 
feed these results into a final model 
that makes the prediction based on 
the other models’ predictions 

e fit seems like we are approaching 
a neural network...you are correct! 
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Level 1 
Predictions 
* C1, C2, and C3 are considered 
level 1 classifiers. 


Strategy 


- Many models naturally overfit 
- Randomization — New models 
- New models overfit in different ways 


- Aggregation — Smooth over different ways 
of overfitting to reduce variance 
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Bagging 


Target Population 





Synchronic Aggregation 


- Bootstrap Aggregating. 


- Algorithm to repeat many times: 
a. Create a sample from your data. 
b. Train a model (e.g. a decision tree) on 
that sample. 


- Final model comes by averaging over those 
many models. 


Random Forests 


- We already have: Level 1 of Randomization 
o Train each model on random sample 


- Add: Level 2 of Randomization 
o Choose a random set of features at 
each decision point 





Extremely Randomized Trees 


We already have: Level 1 and Level 2 of 
Randomization 
o Train each model on random sample 
o Choose a random set of features at 
each decision point 


Add: Level 3 of Randomization 
o Choose a path at random! 





Boosting 





Strategy 


- Prevent overfitting from start 
- Train an underfit model m 


- Update m by training a new model on the 
residuals of m 


Diachronic Learning 


Boosting 


- Prevent overfitting from start 
- Train a bad model m 


- Improve m by making quantitative use of the 
residuals of m 





